Revisiting Softmax Masking for Stability in Continual Learning

In continual learning, many classifiers use softmax function to learn confidence. However, numerous studies have pointed out its inability to accurately determine confidence distributions for outliers, often referred to as epistemic uncertainty. This inherent limitation also curtails the accurate decisions for selecting what to forget and keep in previously trained confidence distributions over continual learning process. To address the issue, we revisit the effects of masking softmax function. While this method is both simple and prevalent in literature, its implication for retaining confidence distribution during continual learning, also known as stability, has been under-investigated. In this paper, we revisit the impact of softmax masking, and introduce a methodology to utilize its confidence preservation effects. In class- and task-incremental learning benchmarks with and without memory replay, our approach significantly increases stability while maintaining sufficiently large plasticity. In the end, our methodology shows better overall performance than state-of-the-art methods, particularly in the use with zero or small memory. This lays a simple and effective foundation of strongly stable replay-based continual learning

Feature Structure Distillation for BERT Transferring

Knowledge distillation is an approach to transfer information on representations from a teacher to a student by reducing their difference. A challenge of this approach is to reduce the flexibility of the student's representations inducing inaccurate learning of the teacher's knowledge. To resolve it in BERT transferring, we investigate distillation of structures of representations specified to three types: intra-feature, local inter-feature, global inter-feature structures. To transfer them, we introduce \textit{feature structure distillation} methods based on the Centered Kernel Alignment, which assigns a consistent value to similar features structures and reveals more informative relations. In particular, a memory-augmented transfer method with clustering is implemented for the global structures. In the experiments on the nine tasks for language understanding of the GLUE dataset, the proposed methods effectively transfer the three types of structures and improve performance compared to state-of-the-art distillation methods. Indeed, the code for the methods is available in https://github.com/maroo-sky/FSDComment: This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessibl

Dynamic Interactions between Carbon and Energy Prices in the U.S. Regional Greenhouse Gas Initiative Region

Numerous studies have investigated the dynamic interrelationship between carbon emission trading market and energy markets. Previous studies focused on the European Union Emissions Trading Scheme ascertain that carbon market and energy markets are closely attached, and find that electricity market is the main driver of the system. Our research on U.S. Regional Greenhouse Gas Initiative (RGGI) using Lag Augmented Vector Autoregression reveals that the RGGI market and electricity market in the region are tied but not strongly, unlike the EU-ETS. This loose relationship between the two markets might be explained by the recent weak carbon credit demand stemming from fuel switching and low electricity demand. Another finding is that natural gas is the main driver of the RGGI system, which is possibly due to from the recent shale gas boom. Keywords: Carbon emission trading; Lag Augmented Vector Autoregression; Regional Greenhouse Gas Initiative  JEL Classifications: C32; Q52; Q5

Stretchable, Patch-Type, Wireless, 6-axis Inertial Measurement Unit for Mobile Health Monitoring

Wearable devices which measure and transfer signals from the human body can provide useful biometric data for various biomedical applications. In this paper, we present an implementation of the advanced Inertial Measurement Unit (IMU) with wireless communication technology for mobile health monitoring. The device consists of rigid silicon-based components on a flexible/stretchable substrate for applications in epidermal electronic devices to collect precise data from the human body. Using the Bluetooth Low Energy (BLE) System-on-a-chip (SoC), the device can be miniaturized and portable, and the collected data can be processed with low power consumption. The dimensions of the implemented system are approximately 40 mm × 40 mm × 100 mm. Also, the device can be attached closely to human skin, which results in minimized signal distortion due to body movements or skin deformations. In order to achieve device flexibility and stretch ability, the interconnection wires are designed as serpentine-shaped structures on a stretchable substrate. The previously reported “cut-and-paste” method is utilized to fabricate the device that produces complex, twisty interconnections with thin metal sheets. The implemented patch-type, wireless, 6-axis IMU is expected to have potential in various applications, such as health monitoring, dependency care, and daily lifelogging

Participation “In the Heavenlies” in Christ: Deification in Ephesians

Paul’s expression “in the heavenlies” provides an intriguing showcase of the power dynamics of the divine–human relationship (e.g., 1:3, 20; 2:6; 3:10; 6:12). While scholars have identified the theme of union with Christ as an interpretive key for understanding believers’ position in the heavenlies, few have provided adequate attention to “in the heavenlies” according to the significance of theosis. I argue that a patristic idea of theosis offers an interpretive lens in understanding believers’ lives in the heavenlies. Thus, this study aims to situate the discussion on the heavenlies vis-à-vis the conversation around theosis in the New Testament

A Novel Top-Down Fabrication Process for Vertically-Stacked Silicon-Nanowire Array

Silicon nanowires are widely used for sensing applications due to their outstanding mechanical, electrical, and optical properties. However, one of the major challenges involves introducing silicon-nanowire arrays to a specific layout location with reproducible and controllable dimensions. Indeed, for integration with microscale structures and circuits, a monolithic wafer-level process based on a top-down silicon-nanowire array fabrication method is essential. For sensors in various electromechanical and photoelectric applications, the need for silicon nanowires (as a functional building block) is increasing, and thus monolithic integration is highly required. In this paper, a novel top-down method for fabricating vertically-stacked silicon-nanowire arrays is presented. This method enables the fabrication of lateral silicon-nanowire arrays in a vertical direction, as well as the fabrication of an increased number of silicon nanowires on a finite dimension. The proposed fabrication method uses a number of processes: photolithography, deep reactive-ion etching, and wet oxidation. In applying the proposed method, a vertically-aligned silicon-nanowire array, in which a single layer consists of three vertical layers with 20 silicon nanowires, is fabricated and analyzed. The diamond-shaped cross-sectional dimension of a single silicon nanowire is approximately 300 nm in width and 20 μm in length. The developed method is expected to result in highly-sensitive, reproducible, and low-cost silicon-nanowire sensors for various biomedical applications

Pano-AVQA: Grounded Audio-Visual Question Answering on 360◦ Videos

© 2021 IEEE360◦ videos convey holistic views for the surroundings of a scene. It provides audio-visual cues beyond predetermined normal field of views and displays distinctive spatial relations on a sphere. However, previous benchmark tasks for panoramic videos are still limited to evaluate the semantic understanding of audio-visual relationships or spherical spatial property in surroundings. We propose a novel benchmark named Pano-AVQA as a large-scale grounded audio-visual question answering dataset on panoramic videos. Using 5.4K 360◦ video clips harvested online, we collect two types of novel question-answer pairs with bounding-box grounding: spherical spatial relation QAs and audio-visual relation QAs. We train several transformer-based models from Pano-AVQA, where the results suggest that our proposed spherical spatial embeddings and multimodal training objectives fairly contribute to a better semantic understanding of the panoramic surroundings on the dataset.N